Video stream management for remote graphical user interfaces

ABSTRACT

Embodiments enable display updates other than a video stream in a graphical user interface (GUI) to be rendered, encoded, and transmitted exclusive of the video stream. A virtual machine generates a GUI that includes an encoded video stream and other display updates. A virtual graphics processing unit (VGPU) stack associated with the VM renders the other display updates of the GUI to a framebuffer. The rendered display updates are encoded and transmitted to a client for display. The encoded video stream, or a modified (e.g., reduced bit rate) version of the encoded video stream, may be transmitted to the client, such that the client can display the encoded video stream within the GUI. For example, the encoded video stream may be selectively transmitted to the client based on the performance capabilities of the client.

CROSS REFERENCE TO RELATED APPLICATION

This application is a continuation application of U.S. patentapplication Ser. No. 15/167,946, filed on May 27, 2016, which is acontinuation application of U.S. patent application Ser. No. 13/666,077,filed Nov. 1, 2012, now U.S. Pat. No. 9,372,737, which claims thebenefit of U.S. Provisional Patent Application No. 61/596,561 filed Feb.8, 2012, all of which are incorporated herein in its entirety.

BACKGROUND

Remote desktop technology allows a user of a client computing device tointeract with a graphical user interface (e.g., a desktop interface)that is managed by a server computing device. The graphical userinterface (GUI) includes display information to be presented on adisplay for a user. For example, display updates generated by softwareapplications executed by the server may be rendered to a framebuffer atthe server and then transmitted to the client for display. Theframebuffer stores pixel color values for every pixel for an outputdisplay. Remote desktop and computing applications export the GUI to aremote device, referred to herein as the “client device.” Conversely,user inputs (e.g., mouse clicks and keystrokes) at the client device maybe transmitted to the server for input into applications or othersoftware running at the server.

To limit the amount of data transferred between the server and client,the GUI may be encoded. For example, the server may compress every frameand/or calculate differences between a current rendering (e.g., frame)of the desktop interface and a previous rendering, and transmit thedifferences to the client, rather than transmitting the entirety of thedesktop interface.

Further, the desktop interface may be “virtual,” generated by a virtualmachine (VM) executed by the server. In such virtual desktopinfrastructures (VDIs), the GUI may be rendered by a virtual graphicsprocessing unit (VGPU) stack associated with the VM, encoded, andtransmitted to the client for display. When a video stream provided inencoded form is to be displayed within the desktop interface, the VGPUdecodes the encoded video stream and renders the decoded video streamwithin the desktop interface. The desktop interface, including thedecoded video stream, is then encoded and transmitted to the client.

SUMMARY

The methods and systems described herein enable display updates otherthan a video stream in a graphical user interface (GUI), such as avirtual desktop, to be transmitted to a client exclusive of the videostream. In exemplary embodiments, a VM executed by a server generates aGUI that includes an encoded video stream and other display updates. Avirtual graphics processing unit (VGPU) stack associated with the VMrenders the other display updates of the GUI to a framebuffer withoutdecoding or rendering the video stream to the framebuffer. The serverencodes the rendered display updates and transmits the encoded displayupdates to a client for display. In some embodiments, the encoded videostream, or a modified (e.g., reduced bit rate) version of the encodedvideo stream, is transmitted to the client using a communication channelother than the communication channel used to transmit the encoded GUI,such that the client can decode and display the encoded video streamwithin the GUI.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram of a system for providing access to agraphical user interface.

FIG. 2 is a flowchart of an exemplary method for providing access to agraphical user interface using the system shown in FIG. 1.

DETAILED DESCRIPTION

Embodiments described herein facilitate reducing server computing loadin a virtual desktop infrastructure (VDI) by rendering display updatesto a graphical user interface (GUI) other than a video stream, andencoding and transmitting the rendered display updates, exclusive of thevideo stream, to a client. Accordingly, the encoded video stream neednot be decoded and rendered within the GUI by the server, potentiallyreducing the volume (e.g., the quantity and/or the size) of displayupdates that are encoded and transmitted to the client. As a result, theutilization of processor and/or network resources used to encode andtransmit such display updates may also be reduced.

In some embodiments, the encoded video stream is discarded by theserver. In other embodiments, the server transmits the encoded videostream to the client separately from other display updates in the GUI ormodifies (e.g., reduces the bit rate of) the encoded video stream andtransmits the modified version of the encoded video stream to the clientseparately from other display updates in the GUI. For example, theserver may determine the performance capabilities (e.g., networkbandwidth or a hardware attribute) of the client and determine whetherto prevent transmission of the encoded video stream, transmit theoriginal encoded video stream, or transmit a modified version of theencoded video stream based on the performance capabilities.

FIG. 1 is a block diagram of an exemplary system 100 for providingremote access to a GUI, such as a virtual desktop interface. Inexemplary embodiments, a server computing device 105 (“server”) executesa virtual machine (VM) 110 that generates a GUI for display at a clientcomputing device 115 (“client”). For example, the GUI may include adesktop interface produced by a guest operating system executed withinVM 110. In addition, or alternatively, the GUI may include graphicalelements (e.g., a window) produced by a software application executed bysuch a guest OS.

In exemplary embodiments, VM 110 generates the GUI by instructing avirtual graphics processing unit (VGPU) stack 120 that is associatedwith VM 110 to draw and display graphical elements, such as windows,fields, icons, images, and input controls (e.g., buttons), of the GUI.The instructions to the VGPU may be referred to as “display updates.”VGPU stack 120 includes, for example, a graphics driver (not shown),which may be included in VM 110, a software-emulated or “virtual”graphics device (not shown), such as a virtual graphics adapter, and avirtual display device (not shown), such as a virtual monitor (e.g.,liquid crystal display). Software applications executed by VM 110interact with the graphics driver of VGPU stack 120, either directly orvia the guest OS. The virtual graphics device interacts with guestsoftware executing in the VM as would a physical graphics device.

VGPU stack 120 renders the display updates in the GUI as instructed byVM 110. In exemplary embodiments, VGPU stack 120 renders the displayupdates by drawing a frame of pixels depicting the graphical elements toone or more framebuffers (not shown) provided by the virtual graphicsdevice (not shown). Successive frames may be drawn by updating theframebuffer(s), such as by drawing successive frames to alternatingframebuffers, or “multiple buffering.”

A GUI encoder 125 encodes the rendered display updates and transmits theencoded display updates to client 115. The display updates may beencoded, for example, by applying a lossless or lossy data compressionalgorithm to a drawn frame, or to modified pixel values within theframe, and/or by applying video compression to a sequence of framesdrawn by VGPU stack 120. Video compression may include interframecompression, such as encoding a given frame as a set of differencesbetween that frame and a previous frame.

Client 115 receives the encoded display updates and decodes the encodeddisplay updates using a GUI decoder 130. The decoded display updates aredisplayed at a view screen 135 of client 115 to update the GUI asinstructed by VM 110, such as by rendering the decoded display updatesto a framebuffer of a physical graphics device that is connected to aphysical display device.

In some scenarios, a video stream is displayed within the GUI. Forexample, a user of client 115 may access a video stored at server 105using a local video playback application 140, or may access a videohosted on a network (e.g., the World Wide Web) using a network streamingapplication 145, both of which are executed at server 105 by VM 110.Such videos are typically provided in the form of an encoded videostream, which enables the video to be compressed using a video-specificcompression algorithm (e.g., including interframe compression). Forexample, a video stream may be compressed using the H.264 standard forvideo compression. Accordingly, displaying the video stream using theprocess described above may involve decoding the encoded video stream,generating display updates based on the video stream, and rendering thedisplay updates to the framebuffer using VGPU stack 120. Such animplementation would trigger a display update in the GUI each timesuccessive frames of the video stream differ, even if nothing within theGUI other than the video stream was modified. As a result, GUI encoder125 would encode and transmit the display update for each such change,essentially re-encoding the video stream. In addition to imposing acomputing load on server 105, such an implementation may provide a videostream that client 115 is incapable of displaying at an acceptablequality level (e.g., frame rate) due to performance limitations ofclient 115.

FIG. 2 is a flowchart of an exemplary method 200 for providing access toa GUI using system 100 (shown in FIG. 1). Referring to FIGS. 1 and 2, asdescribed above, server 105 executes 205 VM 110. As VM 110 executes,VGPU stack 120 receives 210 a request to display a video stream (e.g.,from local video playback application 140 or network streamingapplication 145) within the GUI. The video stream is provided (e.g.,stored locally or streamed from another computing device) in encodedform.

VGPU stack 120 renders 220 display updates in the GUI other than thevideo stream to the framebuffer. VGPU stack 120 may or may not renderthe video stream, depending on a configuration of server 105 and/or aconfiguration provided by client 115, such as a configuration parameterindicating that client 115 supports “video redirection” (e.g., receivingthe encoded video stream in a communication channel separate from thatused to transmit 230 other display updates).

When the configuration specifies that the video stream should not berendered by VGPU stack 120, VGPU stack 120 keeps the encoded videostream separate from other display updates in the GUI. For example, VGPUstack 120 may substitute 222 a placeholder for the encoded video streamwithin the GUI. In some embodiments, VGPU stack 120 substitutes 222 aplaceholder for the video stream by rendering the area of the GUIoccupied by the video stream as a solid color and/or as a textualmessage or image indicating that the encoded video stream has beensuppressed. GUI encoder 125 encodes 225 the rendered display updates,which exclude the video stream, and transmits 230 the encoded displayupdates to client 115, as described above with reference to FIG. 1.

When the configuration specifies that the video stream should berendered by VGPU stack 120, VGPU stack 120 decodes and renders 227 thevideo stream to the framebuffer with the other display updates. GUIencoder 125 encodes 228 the rendered display updates, including thevideo stream. Optionally, GUI encoder 125 may exclude the video streamand/or substitute a placeholder for the video stream, as describedabove, when encoding 228 the rendered display updates. For example, GUIencoder 125 may determine whether to encode the video stream based on aconfiguration parameter provided by client 115. Performing suchexclusion and/or substitution at GUI encoder 125 enables system 100 toaccommodate scenarios in which video redirection capabilities changeduring the computing session in which the GUI is presented. For example,if client 115 becomes unable to support video redirection due to amalfunction, client 115 may request that GUI encoder 125 include thevideo stream when encoding 228 display updates. Similarly, if thecomputing session is disconnected from a first client that supportsvideo redirection and reconnected to a second client that does notsupport video redirection, the second client may request that GUIencoder 125 include the video stream when encoding 228 display updates.In response to such a request, GUI encoder 125 encodes 228 the videostream along with the other display updates and transmits 230 theencoded display updates to client 115.

In exemplary embodiments, rendering 220 and encoding 225 display updatesother than the video stream for transmission 230 to client 115 avoidsdecoding of the video stream by VGPU stack 120 and re-encoding of thedecoded video stream as display updates within the GUI by GUI encoder125. Accordingly, the utilization of computing resources (e.g.,processor cycles and/or network bandwidth) associated with providing theGUI to client 115 may be significantly reduced. Such a computingresource utilization reduction may be especially pronounced where server105 provides GUIs to a plurality of clients 115. In some embodiments,VGPU stack 120 decodes and renders 227 the video stream to theframebuffer, but GUI encoder 125 excludes the video stream when encoding228 display updates. Such embodiments incur some computing resource(e.g., processor cycles) utilization due to the decoding and rendering227 but still exhibit reduced computing resource (e.g., processor cyclesand network bandwidth) utilization by avoiding encoding and transmissionof display updates based on the video stream.

Further, where client 115 is a device with limited computing resources,such as a smart phone or a device with limited network bandwidth, theencoded video stream may not be displayable at client 115 at anacceptable level of quality. For example, client device 115 may displayonly a portion of the frames in the video stream (also known as“dropping” frames), potentially causing the video stream to beunintelligible. In such scenarios, omitting the encoded video streamfrom display updates transmitted 230 to client 115 may not negativelyaffect the computing experience of a user at client 115.

Exemplary embodiments omit decoding and rendering 227 of the videostream using VGPU stack 120, at the level of a graphics driver or avirtual device. For example, VGPU stack 120 may, when queried forcapabilities by the guest OS of VM 110, advertise hardware accelerationof one or more video decoding operations (e.g., motion compensation,de-interlacing, decryption, post-processing, and/or supportedencodings). Accordingly, software applications, such as local videoplayback application 140 and network streaming application 145, maydelegate decoding and rendering of an encoded video stream to VGPU stack120 using a standard, general-purpose interface that supports hardwareacceleration, such as DirectX Video Acceleration (DXVA), provided byMicrosoft Windows™. Accordingly, the video stream may be decoded andrendered 227, or omitted, as determined by VGPU stack 120. Suchembodiments enable server 105 to accommodate video streams provided byany software application, rather than accommodating only applicationsthat use a specialized application programming interface (API).

In some embodiments, the encoded video stream is not provided to client115. In other embodiments, the encoded video stream is optionallytransmitted to client 115. In such embodiments, an intercept module 150executed by server 105 captures the encoded video stream from VGPU stack120 using a stream capture component 155. A control module 160 ofintercept module 150 selects 235 one of three available transmissionactions: a) preventing transmission of the encoded video stream, b)transmitting the encoded video stream, and c) transmitting a modifiedversion of the encoded video stream.

As one example, control module 160 may be configured to always select235 a predetermined transmission action. As another example, interceptmodule 150 may be configured to select 235 a transmission action basedon a preference or configuration associated with VM 110, with client115, and/or with the user of client 115. In addition, or alternatively,control module 160 may select 235 a transmission action based on clientperformance capabilities (e.g., provided by a capabilities module 165),as described in more detail below.

Intercept module 150 performs the selected transmission action. Whenprevention of transmission is selected 235, intercept module 150discards 240 the encoded video stream, such that the encoded videostream is not transmitted to client 115. When transmission of theencoded video stream is selected 235, intercept module 150 transmits 245the original encoded video stream, as received from VGPU stack 120, toclient 115. In exemplary embodiments, GUI encoder 125 transmits 230 theencoded display updates to client 115 using a first communicationchannel (e.g., a first network socket), and intercept module 150transmits 245 the encoded video stream to client 115 using a secondcommunication channel (e.g., a second network socket).

Client 115 receives the encoded video stream and decodes the encodedvideo stream using a video decoder 170. Video decoder 170 may include ahardware decoder and/or a software decoder executed by a processor ofclient 115. For example, some examples of client 115, such as a netbook,may not include a hardware decoder. Other examples of client 115, suchas a desktop computer, may include a hardware decoder (e.g., a graphicsprocessing unit, GPU) capable of performing at least some video decodinghardware acceleration. Whether implemented as hardware, software, orboth, video decoder 170 decodes the encoded video stream, and client 115renders the decoded video stream within the GUI using a video overlaycomponent 175, such that the decoded video stream is displayed withinthe GUI to the user of client 115.

When transmission of a modified version of the encoded video stream isselected 235, prior to transmitting 245 the video stream, interceptmodule 150 modifies 250 the encoded video stream using a transcoder 180.In exemplary embodiments, transcoder 180 modifies the encoded videostream's encoding, bit rate, frame rate, and/or resolution. For example,transcoder 180 may reduce the bit rate of the encoded video stream bydecoding the video stream and re-encoding the decoded video stream usinga video compression algorithm that achieves a higher compression ratiothan that achieved by the compression algorithm used to encode theoriginal encoded video stream, and/or using video compression parametersthat allow a greater amount of data loss than allowed by the parametersused to encode the original encoded video stream. As another example,transcoder 180 may decode the video stream, scale the decoded videostream to a lower resolution than the resolution of the original encodedvideo stream, and encode the scaled video stream. Such modification 250enables server 105 to reduce the network bandwidth used to transmit 245the encoded video stream to client 115, and/or to reduce the computingload incurred by client 115 in decoding the encoded video stream.

In exemplary embodiments, control module 160 also transmits 255 controlinformation associated with the video stream and/or with the GUI toclient 115. The control information includes video stream attributes(e.g., the video compression format, the resolution, the frame rate,and/or the bit rate of the video stream), one or more dimensions (e.g.,the width and/or height) of the video stream, the position (e.g., anx-coordinate, a y-coordinate, and/or a z-coordinate) of the video streamwithin the GUI, and/or compositing information associated with the videostream and the GUI (e.g., which graphical elements, if any, underlay oroverlay the video stream in the GUI). Video overlay component 175 ofclient 115 renders the decoded video stream within the GUI based on thecontrol information. For example, if other graphical elements in the GUIoverlap the video stream, video overlay component 175 may overlay thevideo stream on the graphical elements that are positioned below thevideo stream (e.g., elements having a lower z-coordinate than that ofthe video stream) and overlay the graphical elements that are positionedabove the video stream (e.g., elements having a higher z-coordinate thanthat of the video stream) on the video stream. Accordingly, aside fromany modification 250 performed by transcoder 180, the video stream isdisplayed in view screen 135 of client 115 as the video stream would bedisplayed to a local user at server 105, but without incurring thecomputing resource utilization of decoding the video stream andre-encoding the video stream within the GUI at server 105.

As described above, in some embodiments, VGPU stack 120 and/or GUIencoder 125 provides a message and/or an image indicating that the videostream has been suppressed. When the encoded video stream is transmitted245 to client 115, video overlay component 175 may overlay the videostream on such an indication, whereas when the encoded video stream isdiscarded 240, the indication is shown. In other embodiments, no suchindication is provided by server 105, and when server 105 discards 240the encoded video stream, video overlay component 175 renders a messageor image indicating the suppression of the video stream at the position(e.g., as indicated by the control information) of the video streamwithin the GUI. For example, video overlay component may display amessage such as “Video stream suppressed at remote desktop.”

As described above, control module 160 may select 235 a transmissionaction based on a configuration of control module 160, VM 110, client115, and/or the user of client 115. In some embodiments, thetransmission action is selected 235 based on one or more performancecapabilities of client 115. In such embodiments, capabilities module 165determines 232 client performance capabilities, such as a networkbandwidth of client 115, a hardware attribute of client 115 (e.g.,processor speed, amount of memory, and/or presence of a hardware videodecoder 170), hardware acceleration of one or more video decodingoperations (e.g., motion compensation, de-interlacing, decryption,post-processing, and/or supported encodings), a resolution of client115, and/or a frame rate of video displayed at client 115. For example,the client performance capabilities may be stored as a profileassociated with client 115 and may be manually or automatically defined.

In some embodiments, capabilities module 165 determines 232 clientperformance capabilities at least in part by receiving the capabilitiesfrom client 115. In the embodiment shown in FIG. 1, capabilities module165 receives the client performance capabilities via a view agent 185that communicates with client 115. For example, view agent 185 mayoperate as a proxy for client 115, such that capabilities module 165 canrequest and receive capabilities of client 115 from view agent 185,which relays requests and responses between capabilities module 165 andclient 115. In addition, or alternatively, client 115 may initiate atransmission of its performance capabilities to capabilities module 165,for example, when initializing a remote desktop session with server 105.

Control module 160 receives the performance capabilities fromcapabilities module 165 and selects 235 a transmission option based onthe performance capabilities. As an example, control module 160 mayselect 235 preventing transmission of the encoded video stream, ormodifying the encoded video stream (e.g., to reduce the bit rate), ifclient 115 does not include a hardware video decoder.

In exemplary embodiments, control module 160 selects 235 a transmissionoption by comparing a client performance capability to a predeterminedthreshold value. For example, if the network bandwidth of client 115(e.g., the bandwidth from server 105 to client 115) is less than aminimum video streaming threshold value (e.g., 500 kilobits per second,kbps), control module 160 may select 235 preventing transmission of theencoded video stream, such that the video stream does not interfere withdisplay of the GUI. If the network bandwidth is greater than the minimumvideo streaming threshold value and less than a moderate video streamingthreshold value (e.g., 1000 kbps), control module 160 may select 235modifying the encoded video stream to reduce the bit rate, such that thevideo stream may be transmitted to client 115 without interfering withthe display of the GUI. If the network bandwidth is greater than themoderate video streamlining threshold value, control module 160 mayselect 235 transmitting the original encoded video stream.

Threshold values may be predetermined, as described above, or may bedetermined by control module 160 based on attributes of the encodedvideo stream. In an exemplary embodiment, control module 160 selects 235the transmission option in part by calculating a minimum video streamingthreshold value based on the bit rate of the encoded video streamreceived by stream capture component 155. For example, the thresholdvalue may be equal to the sum of the bit rate of the encoded videostream, an estimated average bandwidth associated with transmitting 230display updates in the GUI, and, optionally, a predetermined bandwidthoverhead value representing an estimated amount of bandwidth used byclient 115 for purposes other than displaying the GUI and/orrepresenting an estimated amount by which the available bandwidth mayfluctuate. In another embodiment, control module 160 calculates adesired resolution threshold value equal to the resolution of theencoded video stream received by stream capture component 155.

In some embodiments, control module 160 selects 235 the transmissionoption based on a plurality of performance capabilities. For example, afirst set of threshold values (e.g., minimum processor speed) may beapplied when client 115 includes a hardware video decoder 170, and asecond set of threshold values may be applied when client 115 does notinclude a hardware video decoder 170.

In one embodiment, when control module 160 selects 235 modifying thevideo stream, transcoder 180 modifies 250 the encoded video stream basedon a predetermined set of target attributes, such as a target bit rate,a target frame rate, a target resolution, and/or a target encoding. Forexample, whenever modification is selected 235, transcoder 180 maymodify 250 the encoded video stream by decoding the encoded video streamand re-encoding the decoded video stream using a predeterminedresolution and/or a predetermined bit rate expected to enable client 115to decode and display the video stream.

In another embodiment, transcoder 180 modifies 250 the encoded videostream based on the performance capabilities of client 115, such as bydetermining 248 (e.g., calculating) one or more target attributes forthe video stream based on the performance capabilities, and modifying250 the encoded video stream using the determined target attributes. Asone example, transcoder 180 may determine 248 a target encoding from aset of encodings supported by client 115 and re-encode the video streamusing the target encoding. As another example, transcoder 180 maydetermine 248 a target resolution that is equal to the resolution ofclient 115 and modify 250 the encoded video stream by scaling the videostream such that the resolution of the transmitted video stream is lessthan or equal to the target resolution. As yet another example,transcoder 180 may reduce the bit rate of the video stream to a targetbit rate that is based on the processor speed and/or bandwidth of client115.

In some embodiments, transcoder 180 determines 248 a target attribute byevaluating a mathematical function (e.g., a stepwise or continuousfunction) using one or more client capabilities as input values. Forexample, such a function may provide a target bit rate that ispositively correlated with the processor speed and/or the bandwidth ofclient 115.

Some performance capabilities, such as processor speed and the presenceof a hardware video decoder 170, are static. Other performancecapabilities, such as bandwidth and frame rate, may be dynamic.Accordingly, the process of determining 232 client performancecapabilities and selecting 235 a transmission action may be performedrepeatedly, such that control module 160 can adapt the transmissionaction as the client performance capabilities change over time. Forexample, if client 115 is a mobile device, the bandwidth of client 115may change significantly as client 115 moves out of range a wirelesslocal area network (WLAN) and begins communicating with server 105 usinga mobile data (e.g., 3G or 4G) network. Similarly, client 115 may detectand report to capabilities module 165 (via view agent 185) the framerate of encoded video stream as it is decoded and rendered to viewscreen 135. If the detected frame rate is less than the frame rate ofthe encoded video stream transmitted 245 to client 115, control module160 may select 235 modifying the video stream to reduce the bit rate.

Exemplary Operating Environment

The operations described herein may be performed by a computer orcomputing device. A computer or computing device may include one or moreprocessors or processing units, system memory, and some form of computerreadable storage media. Computer-readable storage media are tangible andnon-transitory and store information such as computer readableinstructions, data structures, program modules, or other data. Exemplarycomputer readable storage media include hard disk drives, flash memorydrives, digital versatile discs (DVDs), compact discs (CDs), andmagnetic tape cassettes.

The operations illustrated and described herein may be implemented ascomputer-executable instructions encoded on a computer-readable medium,in hardware programmed or designed to perform the operations, or both.Aspects of the disclosure transform a general-purpose computer into aspecial-purpose computing device when programmed to execute theoperations described herein.

The order of execution or performance of the operations in embodimentsof the disclosure illustrated and described herein is not essential,unless otherwise specified. That is, the operations may be performed inany order, unless otherwise specified, and embodiments of the disclosuremay include additional or fewer operations than those disclosed herein.For example, it is contemplated that executing or performing aparticular operation before, contemporaneously with, or after anotheroperation is within the scope of aspects of the disclosure.

When introducing elements of aspects of the disclosure or theembodiments thereof, the articles “a,” “an,” “the,” and “said” areintended to mean that there are one or more of the elements. The terms“comprising,” “including,” and “having” are intended to be inclusive andmean that there may be additional elements other than the listedelements.

Having described aspects of the disclosure in detail, it will beapparent that modifications and variations are possible withoutdeparting from the scope of aspects of the disclosure as defined in theappended claims. As various changes could be made in the aboveconstructions, products, and methods without departing from the scope ofaspects of the disclosure, it is intended that all matter contained inthe above description and shown in the accompanying drawings shall beinterpreted as illustrative and not in a limiting sense.

What is claimed is:
 1. A method for providing a graphical user interface(GUI) from a server computing device to a client device, the methodcomprising: executing, by the server computing device, a virtual machine(VM) that generates GUI elements of the GUI associated with the VM fordisplay on a client device; receiving a request to display a videostream within the GUI; rendering, on the server, display elements of theGUI other than the requested video stream including a placeholderdisplay element for an area of the GUI occupied by the video stream andtransmitting the rendered display elements to the client device fordisplay; and transmitting the video stream to the client device, whereinthe video stream is configured for decoding by the client device andrendering within the GUI presented on the client device.
 2. The methodof claim 1, wherein the display elements are transmitted to the clientdevice in a first communication channel and the encoded video stream istransmitted separately from the rendered display elements using a secondcommunication channel.
 3. The method of claim 1, further comprising:transmitting control information associated with the video stream to theclient device.
 4. The method of claim 1, further comprising:determining, by the server, a performance capability of the clientdevice; and modifying the video stream based on the determinedperformance capabilities prior to transmitting the video stream to theclient device.
 5. The method of claim 1, wherein the rendered GUIdisplay elements are encoded prior to transmission, the encodingincluding applying compressing the display elements.
 6. The method ofclaim 1, wherein a virtual graphics processing unit (VGPU) stack rendersthe display elements of the GUI while keeping the video stream separatefrom the display elements.
 7. A system comprising: a server computingdevice including one or more processors and one or more storage deviceshaving computer-executable components including instructions that whenexecuted perform operations comprising: executing, by the servercomputing device, a virtual machine (VM) that generates GUI elements ofthe GUI associated with the VM for display on a client device; receivinga request to display a video stream within the GUI; rendering, on theserver, display elements of the GUI other than the requested videostream including a placeholder display element for an area of the GUIoccupied by the video stream and transmitting the rendered displayelements to the client device for display; and transmitting the videostream to the client device, wherein the video stream is configured fordecoding by the client device and rendering within the GUI presented onthe client device.
 8. The system of claim 7, wherein the displayelements are transmitted to the client device in a first communicationchannel and the encoded video stream is transmitted separately from therendered display elements using a second communication channel.
 9. Thesystem of claim 7, further including instructions that when executedperform operations comprising: transmitting control informationassociated with the video stream to the client device.
 10. The system ofclaim 7, further including instructions that when executed performoperations comprising: determining, by the server, a performancecapability of the client device; and modifying the video stream based onthe determined performance capabilities prior to transmitting the videostream to the client device.
 11. The system of claim 7, wherein therendered GUI display elements are encoded prior to transmission, theencoding including applying compressing the display elements.
 12. Thesystem of claim 7, wherein a virtual graphics processing unit (VGPU)stack renders the display elements of the GUI while keeping the videostream separate from the display elements.
 13. One or morenon-transitory computer-readable storage media havingcomputer-executable components including instructions that when executedperform operations comprising: executing, by a server computing device,a virtual machine (VM) that generates GUI elements of the GUI associatedwith the VM for display on a client device; receiving a request todisplay a video stream within the GUI; rendering, on the server, displayelements of the GUI other than the requested video stream including aplaceholder display element for an area of the GUI occupied by the videostream and transmitting the rendered display elements to the clientdevice for display; and transmitting the video stream to the clientdevice, wherein the video stream is configured for decoding by theclient device and rendering within the GUI presented on the clientdevice.
 14. The non-transitory computer-readable storage media of claim13, wherein the display elements are transmitted to the client device ina first communication channel and the encoded video stream istransmitted separately from the rendered display elements using a secondcommunication channel.
 15. The non-transitory computer-readable storagemedia of claim 13, further comprising instructions that when executedperform operations comprising: transmitting control informationassociated with the video stream to the client device.
 16. Thenon-transitory computer-readable storage media of claim 13, furthercomprising instructions that when executed perform operationscomprising: determining, by the server, a performance capability of theclient device; and modifying the video stream based on the determinedperformance capabilities prior to transmitting the video stream to theclient device.
 17. The non-transitory computer-readable storage media ofclaim 13, wherein the rendered GUI display elements are encoded prior totransmission, the encoding including applying compressing the displayelements.
 18. The non-transitory computer-readable storage media ofclaim 13, wherein a virtual graphics processing unit (VGPU) stackrenders the display elements of the GUI while keeping the video streamseparate from the display elements.